Visualization system and method for ent procedures

ABSTRACT

A visualization system may be implemented with an image guided surgery navigation system for use during ENT procedures. Correlated datasets such as surgical paths and pre-operative image slices may be correlated with the coordinate system of the navigation system for reference during a procedure. A surgeon may wear a head mounted display (“HMD”) having a transparent screen through which the patient anatomy may be viewed. The orientation and distance of the head mounted display relative to the patient anatomy may be determined using magnetic position tracking, image analysis of images captured by a camera of the HMD, or both. Correlated datasets may then be transformed based on the relative distance and orientation and may be displayed via the transparent screen to overlay directly viewed patient anatomy. Some implementations may include optical fiducials integrated with a patient tracking assembly to aid in tracking the patient and determining perspective of the HMD.

PRIORITY

This application claims priority to U.S. Provisional Pat. App. No.62/925,441, entitled “Visualization System and Method for ENTProcedures,” filed Oct. 24, 2019, the disclosure of which isincorporated by reference herein.

BACKGROUND

Image guided surgery navigation systems are used during surgicalprocedures to provide additional information and visual perspectives ofa surgical site or other patient anatomy. This may include displayingpre-operative images (e.g., CT images) of the surgical site from variousperspectives, and may also include overlaying markers onto suchdisplayed images to indicate static information, such as a planned paththat a surgical instrument will take during a procedure, as well asdynamic information, such as the present location of a distal tip of thesurgical instrument. Such information may be used to improve theaccuracy and safety with which a surgical instrument is navigated to aparticular location within a patient's body.

Given the breadth of additional information available during imageguided surgery, a surgeon or other practitioner may sometimes becomespatially disoriented while switching between views of the surgicalsite, or while switching between direct viewing of the patient anatomyand viewing of simulated images of the patient anatomy. For example, insome cases a surgeon may be viewing a CT slice of an axial view of thepatient's head on a display of the image guided surgery navigationsystem, while also occasionally viewing the patient's head directly. Insuch cases, the surgeon may become disoriented and unable to determinethe spatial and directional correspondence between their directperceptions and the axial view, which may lead to erroneous movements ofthe surgical instrument within the patient's head.

While several systems and methods have been made and used in ENTprocedures, it is believed that no one prior to the inventors has madeor used the invention described in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments of the invention,and, together with the general description of the invention given above,and the detailed description of the embodiments given below, serve toexplain the principles of the present invention.

FIG. 1 depicts a schematic view of an exemplary surgery navigationsystem being used on a patient seated in an exemplary medical procedurechair;

FIG. 2A depicts a schematic view of an exemplary patient tracker placedon the patient of FIG. 1;

FIG. 2B depicts a perspective view of an exemplary patient tracker, suchas that shown in FIG. 2A;

FIG. 3 depicts a schematic view of exemplary viewing devices usable withthe surgery navigation system of FIG. 1;

FIG. 4 depicts a perspective view of one exemplary viewing device shownin FIG. 3;

FIG. 5 depicts a flowchart of an exemplary set of steps that may beperformed with the surgery navigation system of FIG. 1 to providevisualization for ENT procedures;

FIG. 6A depicts an exemplary interface showing a front view of a patientface that may be provided during visualization for ENT procedures;

FIG. 6B depicts an exemplary interface showing a side view of a patientface that may be provided during visualization for ENT procedures;

FIG. 6C depicts an exemplary interface showing a perspective view of apatient face that may be provided during visualization for ENTprocedures;

FIG. 7A depicts an exemplary computed tomography (CT) image usable withthe surgery navigation system of FIG. 1;

FIG. 7B depicts an exemplary interface showing a front view of a patientface with a CT image overlaid that may be provided during visualizationfor ENT procedures;

FIG. 7C depicts an exemplary interface showing a side view of a patientface with a CT image overlaid with the that may be provided duringvisualization for ENT procedures;

FIG. 7D depicts an exemplary interface showing a perspective view of apatient face with a CT image overlaid that may be provided duringvisualization for ENT procedures; and

FIG. 8 depicts a schematic diagram that illustrates the relativelocations and orientations of a camera and a patient's head within athree-dimensional coordinate system.

The drawings are not intended to be limiting in any way, and it iscontemplated that various embodiments of the invention may be carriedout in a variety of other ways, including those not necessarily depictedin the drawings. The accompanying drawings incorporated in and forming apart of the specification illustrate several aspects of the presentinvention, and together with the description serve to explain theprinciples of the invention; it being understood, however, that thisinvention is not limited to the precise arrangements shown.

DETAILED DESCRIPTION

The following description of certain examples of the invention shouldnot be used to limit the scope of the present invention. Other examples,features, aspects, embodiments, and advantages of the invention willbecome apparent to those skilled in the art from the followingdescription, which is by way of illustration, one of the best modescontemplated for carrying out the invention. As will be realized, theinvention is capable of other different and obvious aspects, all withoutdeparting from the invention. Accordingly, the drawings and descriptionsshould be regarded as illustrative in nature and not restrictive.

For clarity of disclosure, the terms “proximal” and “distal” are definedherein relative to a surgeon, or other operator, grasping a surgicalinstrument having a distal surgical end effector. The term “proximal”refers to the position of an element arranged closer to the surgeon, andthe term “distal” refers to the position of an element arranged closerto the surgical end effector of the surgical instrument and further awayfrom the surgeon. Moreover, to the extent that spatial terms such as“upper,” “lower,” “vertical,” “horizontal,” or the like are used hereinwith reference to the drawings, it will be appreciated that such termsare used for exemplary description purposes only and are not intended tobe limiting or absolute. In that regard, it will be understood thatsurgical instruments such as those disclosed herein may be used in avariety of orientations and positions not limited to those shown anddescribed herein.

As used herein, the terms “about” and “approximately” for any numericalvalues or ranges indicate a suitable dimensional tolerance that allowsthe part or collection of components to function for its intendedpurpose as described herein.

I. Exemplary Image Guided Surgery Navigation System

When performing a medical procedure within a head (H) of a patient (P),it may be desirable to have information regarding the position of aninstrument within the head (H) of the patient (P), particularly when theinstrument is in a location where it is difficult or impossible toobtain an endoscopic view of a working element of the instrument withinthe head (H) of the patient (P). FIG. 1 shows an exemplary IGSnavigation system (10) enabling an ENT procedure to be performed usingimage guidance. In addition to or in lieu of having the components andoperability described herein IGS navigation system (10) may beconstructed and operable in accordance with at least some of theteachings of U.S. Pat. No. 7,720,521, entitled “Methods and Devices forPerforming Procedures within the Ear, Nose, Throat and ParanasalSinuses,” issued May 18, 2010, the disclosure of which is incorporatedby reference herein; and U.S. Pat. Pub. No. 2014/0364725, entitled“Systems and Methods for Performing Image Guided Procedures within theEar, Nose, Throat and Paranasal Sinuses,” published Dec. 11, 2014, nowabandoned, the disclosure of which is incorporated by reference herein.

IGS navigation system (10) of the present example comprises a fieldgenerator assembly (20), which comprises set of magnetic fieldgenerators (24) that are integrated into a horseshoe-shaped frame (22).Field generators (24) are operable to generate alternating magneticfields of different frequencies around the head (H) of the patient (P)to produce a tracked area that the IGS navigation system (10) associatesa coordinate system with. A navigation guidewire (40) is inserted intothe head (H) of the patient (P) in this example. Navigation guidewire(40) may be a standalone device or may be positioned on an end effectoror other location of a medical instrument such as a surgical cuttinginstrument or dilation instrument. In the present example, frame (22) ismounted to a chair (30), with the patient (P) being seated in the chair(30) such that frame (22) is located adjacent to the head (H) of thepatient (P). By way of example only, chair (30) and/or field generatorassembly (20) may be configured and operable in accordance with at leastsome of the teachings of U.S. Pat. No. 10,561,370, entitled “Apparatusto Secure Field Generating Device to Chair,” issued Feb. 18, 2020, thedisclosure of which is incorporated by reference herein.

IGS navigation system (10) of the present example further comprises aprocessor (12), which controls field generators (24) and other elementsof IGS navigation system (10). For instance, processor (12) is operableto drive field generators (24) to generate alternating electromagneticfields; and process signals from navigation guidewire (40) to determinethe location of a sensor in navigation guidewire (40) within the head(H) of the patient (P). Processor (12) comprises a processing unit(e.g., a set of electronic circuits arranged to evaluate and executesoftware instructions using combinational logic circuitry or othersimilar circuitry) communicating with one or more memories. Processor(12) of the present example is mounted in a console (18), whichcomprises operating controls (14) that include a keypad and/or apointing device such as a mouse or trackball. A physician uses operatingcontrols (14) to interact with processor (12) while performing thesurgical procedure.

Navigation guidewire (40) includes a sensor (not shown) that isresponsive to positioning within the alternating magnetic fieldsgenerated by field generators (24). A coupling unit (42) is secured tothe proximal end of navigation guidewire (40) and is configured toprovide communication of data and other signals between console (18) andnavigation guidewire (40). Coupling unit (42) may provide wired orwireless communication of data and other signals.

In the present example, the sensor of navigation guidewire (40)comprises at least one coil at the distal end of navigation guidewire(40). When such a coil is positioned within an alternatingelectromagnetic field generated by field generators (24), thealternating magnetic field may generate electrical current in the coil,and this electrical current may be communicated along the electricalconduit(s) in navigation guidewire (40) and further to processor (12)via coupling unit (42). This phenomenon may enable IGS navigation system(10) to determine the location of the distal end of navigation guidewire(40) or other medical instrument (e.g., dilation instrument, surgicalcutting instrument, etc.) within a three-dimensional space (i.e., withinthe head (H) of the patient (P), etc.). To accomplish this, processor(12) executes an algorithm to calculate location coordinates of thedistal end of navigation guidewire (40) from the position relatedsignals of the coil(s) in navigation guidewire (40). While the positionsensor is located in guidewire (40) in this example, such a positionsensor may be integrated into various other kinds of instruments, suchas dilation catheters, guide catheters, guide rails, suctioninstruments, pointer instruments, registration probes, curettes, patienttrackers, and other instruments, including those described in greaterdetail below.

Processor (12) uses software stored in a memory of processor (12) tocalibrate and operate IGS navigation system (10). Such operationincludes driving field generators (24), processing data from navigationguidewire (40), processing data from operating controls (14), anddriving display screen (16). In some implementations, operation may alsoinclude monitoring and enforcement of one or more safety features orfunctions of IGS navigation system (10). Processor (12) is furtheroperable to provide video in real time via display screen (16), showingthe position of the distal end of navigation guidewire (40) in relationto a video camera image of the patient's head (H), a CT scan image ofthe patient's head (H), and/or a computer generated three-dimensionalmodel of the anatomy within and adjacent to the patient's nasal cavity.Display screen (16) may display such images simultaneously and/orsuperimposed on each other during the surgical procedure. Such displayedimages may also include graphical representations of instruments thatare inserted in the patient's head (H), such as navigation guidewire(40), such that the operator may view the virtual rendering of theinstrument at its actual location in real time. By way of example only,display screen (16) may provide images in accordance with at least someof the teachings of U.S. Pat. No. 10,463,242, entitled “GuidewireNavigation for Sinuplasty,” issued Nov. 5, 2019, the disclosure of whichis incorporated by reference herein. In the event that the operator isalso using an endoscope, the endoscopic image may also be provided ondisplay screen (16).

The images provided through display screen (16) may help guide theoperator in maneuvering and otherwise manipulating instruments withinthe patient's head (H) when such instruments incorporate navigationguidewire (40). It should also be understood that other components of asurgical instrument and other kinds of surgical instruments, asdescribed below, may incorporate a sensor like the sensor of navigationguidewire (40).

In some implementations, the IGS navigation system (10) may include apatient tracking assembly (50) that may be placed on the head (H) of thepatient, or another appropriate portion of the patient (P) as shown inFIG. 2A. By tracking the head (H) separately from the guidewire (40) butwithin the same coordinate system, the positions and orientations of theguidewire (40) and the head (H) may be determined relative to each otherduring a procedure. This may be advantageous where the head (H) isregistered to determine its initial position within the coordinatesystem, but then later moves during the procedure. By tracking the head(H) independently, a rotation or other movement may be detected, and theinitial registration may be updated to account for the new position ofthe head (H). In this manner, any image guided navigation features beingused during the procedure, such as the display of CT image slices withoverlaid markers indicating the position of the guidewire (40) withinthe head (H), may be automatically updated in response to suchmovements. Implementations of the patient tracking assembly (50) may beconstructed and operable with the IGS navigation system (10) inaccordance with any of the teachings of U.S. Pat. Pub. 2019/0183582,entitled “Mounted Patient Tracking Component for Surgical NavigationSystem,” filed Dec. 14, 2017, the disclosure of which is incorporated byreference herein.

As one example of the patient tracking assembly (50), FIG. 2B shows apatient tracking assembly (200) that may be readily incorporated intothe IGS navigation system (10). The patient tracking assembly (200)includes a disposable portion (210) and a reusable portion (250). Thedisposable portion (210) is configured to attach to the patient's head(H), or another suitable portion of patient (e.g., using a flexibleadhesive pad), and is configured to selectively couple with a couplingblock (220) of the reusable portion (250) such that the reusable portion(250) is fixed relative to the head or another portion of patient (P)during exemplary use; while the reusable portion (250) is configured tocommunicate with the processor (12) in order to track the position ofthe head (H) of patient (P).

The reusable portion (250) includes a cable (252) extending proximallyfrom a coupling assembly (254), and a sensor (255). The couplingassembly (254) is adapted to couple the reusable portion (250) with thedisposable portion (210) during use. When properly coupled with thedisposable portion (210), the sensor (255) may be utilized with theprocessor (12) to determine the location of the tracked anatomy, suchthat the processor (12) may accurately display the location of thenavigation guidewire (40) (or any other suitable instrument) relative tothe anatomy of patient (P) during exemplary use. The cable (252) isconfigured to provide a conduit for communication between the sensor(255) and the processor (12) during exemplary use. Therefore, the cable(252) may directly connect such that sensor (255) is in wiredcommunication with the processor (110) via the cable (252).Alternatively, the cable (252) may connect the sensor (255) with awireless communication device that is in wireless communication with theprocessor (12), similar to how the coupling unit (42) establisheswireless communication between the navigation guidewire (40) and theprocessor (12).

II. Exemplary Visualization System for ENT Procedures

FIG. 3 shows a visualization system (60) that may be implemented withthe IGS navigation system (10) in order to provide additionalinformation and image navigation views during a surgical procedure. Inparticular, the visualization system (60) may provide navigation viewsthat display information in an augmented reality view that is overlaidupon images of the patient that are captured during a procedure in nearreal-time. This may advantageously reduce the frequency and need for asurgeon to view the display screen (16) or another display instead ofviewing the patient during a procedure, and may also aid the surgeon inspatially orienting themselves between the physical world and the imagenavigation views provided by the IGS navigation system (10).

The visualization system (60) may be implemented with the IGS navigationsystem (10) by configuring one or more of a head mounted display (“HMD”)(100), a handheld display (“HHD”) (101), or another similar device tocommunicate with the IGS navigation system (10) during an image guidedsurgical procedure, as will be described in more detail below. FIG. 4shows a perspective view of the HMD (100), which includes a frame (102)that may be worn on the face of a surgeon or other user involved withthe image guided surgical procedure. A case (108) is shown mounted tothe frame (102), but may also be worn or mounted elsewhere and coupledwith devices mounted on the frame (102) via a wired or wirelessconnection to those devices. The case (108) includes a perspectivesensor (110) that is operable to provide spatial information indicatingthe perspective, the location, or both of the HMD (100). The perspectivesensor (110) may include one or more of a gyroscope, an accelerometer,or a position sensor (e.g., such as the sensor of the guidewire (40)),for example. Information from the perspective sensor (108) may be usableto determine one or more aspects of a visual perspective of a camera(106) that is mounted to the frame (102), which may include determiningthe orientation of an optical axis the camera (106) (e.g., rotationalcoordinates for one or more axes), the location of the camera (106)relative to the coordinate system of the IGS navigation system (10)(e.g., position coordinates for one more axes), or both, as will bedescribed in more detail below.

The case (108) also includes a communication device (112), which may bea wired or wireless transceiver capable of communicating with theprocessor (12) or other devices, and a processor and memory (114)configured to process and store data and execute functions related tothe function of the HMD (100). A power source (116) is also included,which may be a battery or a connection to an external power source, andwhich is configured to provide power to the processor and memory (114),communication device (112), camera (106), display (104), and othercomponents of the HMD (100).

When worn, the frame (102) positions the camera (106), which is mountedto the frame (102) and/or the case (108), such that its optical axis issubstantially parallel to an optical axis of the wearer's eyes when thewearer looks straight ahead. When used herein, the term “neutral opticalaxis” may refer to the optical axis of a wearer's eye, when the wearerof the HMD (100) looks substantially straight ahead (e.g., where thepupil of the eye is substantially centered both vertically andhorizontally within the orbit or socket of the eye). In this manner, thecamera (106) captures images that have a similar field of view as thatof the wearer of the HMD (100). As an example, the camera (106) maycapture images that include some or all of the field of view of thewearer's right eye, which the camera (106) is positioned mostproximately to, when the wearer is looking straight ahead. The camera(106) may be capable of capturing images, video, and audio, which may bestored by the processor and memory (114), transmitted to the processor(12) via the communication device (112), or transmitted to and displayedor presented on another device, as will be apparent to those skilled inthe art in light of this disclosure. Image data captured by the camera(106) may also be used for computer vision and other analysis which mayinclude, for example, identifying objects or other visualcharacteristics of a captured image. Such analysis may be performed bythe processor (114), the processor (12), or both, or may also beperformed using various cloud computing or edge processing techniques,as will be apparent to those skilled in the art in light of thisdisclosure.

A display (104) is also mounted to the frame (102) and/or the case (108)and is positioned to be within the field of view of the wearer of theHMD (100). In some implementations, the display (104) is at leastpartially translucent if not transparent and is operable by theprocessor (114) to render images that appear to be overlaid upon thefield of view of the wearer. As an example, the display (104) maydisplay an image captured by the camera (106), which may block some orall of the wearer's field of view from the proximate eye, but wouldotherwise appear similar to the wearer's normal field of vision for thateye. As another example, image analysis may be performed on a capturedimage to identify objects of interest within that image (e.g., in thecontext of surgical navigation, a human face or another portion of humananatomy), and the display (104) may be operated to render a visualmarker that appears, to the wearer, to be overlaid upon their directview (e.g., viewed through the transparent portion of the display (104))of the identified object. In some implementations of the above, opticalmarkers or other fiducial markers may be placed on objects of interestin order to provide image data having objects that are easilyidentifiable due to their reflectivity, shape, or other visualcharacteristics, such that an optical marker placed on a human face maybe identified rather than the human face itself.

As yet another example, the display (104) may be operated to rendervisual markings that overlay the wearer's direct view of their field ofview based on other inputs instead of or in addition to image analysisor machine vision. This may include, for example, rendering visualmarkings based on information from the perspective sensor (110), theprocessor (12), the patient tracker (50) (e.g., through communicationwith the processor (12)), and other devices. This could includerendering a visual marking providing information associated with therotational perspective of the HMD (100) (e.g., based on a gyroscopicfeature of the perspective sensor (100)). As another example, this couldinclude rendering a visual marking that overlays a surgical instrument(e.g., the guidewire (40)), based upon tracking information associatedwith the surgical instrument and the HMD (100). In other words, when theprocessor (12) is able to track and determine the relative positions ofthe surgical instrument and the HMD (100), and the orientation of theHMD (100) may be determined (e.g., using the perspective sensor (110)),the position and scale of the tracked objects relative to each other maybe determined and produced as rendered markings via the display (104).

As will be apparent to those skilled in the art in light of thisdisclosure, some or all of the above features may also be performed withother displays beyond the display (104). For example, in someimplementations, a separate display (e.g., the display screen (16) or awall mounted display that is visible to the entire room) may beconfigured to receive and display images captured from the camera (106)and any markings, renderings, or other overlay data that may be added.In such an implementation, the display (104) may render only overlayimages, while the separate display may render a combination of an imagecaptured by the camera (106) and any corresponding overlay images. Thismay allow other participants in the procedure to view the additionalnavigation views in addition to the wearer of the HMD (100). In somesuch implementations, the HMD (100) may not include the display (104),and combinations of captured images and rendered overlays may beviewable on the display screen (16) or another display positioned nearthe patient.

The HHD (101) may share some or all of the capabilities of the HMD(100), and may be, for example, a smartphone, a tablet, a proprietarydevice, or other handheld computing devices having capabilities such asprocessing and storing data, executing functions, sending and receivingdata, capturing images, providing spatial information (e.g.,orientation, location, or both). The display of the HHD (101) maycommonly be a LED or LCD display, and so may not be capable ofoverlaying rendered markings onto a transparent surface through whichobjects are viewed directly, such as the display (104) might. In someimplementations, the HMD (100) or HHD (101) may be modified to includeadditional capabilities. For example, some implementations of the HMD(100) may not include the capability to self-position relative to thecoordinate system of the IGS navigation system (10). In such cases, asensor may be mounted (e.g., externally or internally) on the HMD (100)that allows it to interact with and be tracked by the IGS navigationsystem (10), similar to the guidewire (40) and the patient trackingassembly (50). Thus, the capabilities of the perspective sensor (110)may include both those present in the HMD (100) by default, as well asthose that may be later added, whether they are within the case (108) orexternally mounted on the HMD (100).

In some implementations, information indicating the orientation andlocation of the HMD (100) may be available from multiple sources. As oneexample, the perspective sensor (110) may include a gyroscopic featurecapable of determining rotational orientation, may include orincorporate a tri-axis sensor capable of being tracked by the IGSnavigation system (10) to determine rotational orientation, and may beconfigured to identify optical fiducials present within images capturedby the camera (106). In such examples, the processing components (e.g.,the processor (114), the processor (12), or other processors) may beconfigured to determine orientation or location in various ways bybalancing performance and accuracy, as will be apparent to those skilledin the art in light of this disclosure. For example, someimplementations may determine orientation or location based only ongyroscopic information from the HMD (100) itself or tracking informationfrom the IGS navigation system (10) to emphasize performance, whileother implementations may use a combination of gyroscopic, magnetictracking, and image analysis information with a goal of achieving ahigher accuracy at the potential cost of some performance (e.g., thedelay, if any, between the orientation or location of the HMD (100)changing and a determination of the new orientation or location beingcompleted).

The visualization system (60) allows for additional inputs andinformation to be gathered and used during surgical navigation in orderto provide additional navigation views and other feedback to users ofthe HMD (100) or the HHD (101), or viewers of other displays that areconfigured to display images from the camera (106) and any correspondingoverlay renderings. In the absence of the visualization system (60),there is a breadth of useful information available to the IGS navigationsystem (10) that is largely confined to being used or displayed in thecontext of pre-operative images (e.g., 3-D patient models produced frompre-operative imaging, CT, MRI, or ultrasound image sets).

As an example, the IGS navigation system (10) may allow a surgeon toview a set of CT images for a patient prior to a procedure and plot outa surgical plan or surgical path that the surgeon will navigate one ormore surgical instruments along during the procedure. During theprocedure, the surgical path may then be overlaid on the CT image setand displayed via the display screen (16), allowing the surgeon toswitch between a limited number of views (e.g., axial, coronal,sagittal) and slices as may be desired. As another example, during aprocedure, a surgical instrument (e.g., the guidewire (40)) may also betracked and similarly displayed on the CT image set. When displayedtogether, a surgeon may find it beneficial to view CT images that showthe tracked position of the surgical instrument and the planned surgicalpath in relation to each other.

While useful, the above features can distract the surgeon's attentionfrom the patient, as they may need to look away from the patient to viewa nearby display device. It can also be disorienting when switchingbetween the axial, coronal, and sagittal views, as the surgeon's actuallocation relative to the patient has not changed. For example, a surgeonmay be viewing an axial plane CT image of the patient head and, whenreturning their view to the patient, may be observing a sagittal planeof the patient's head. In order to make use of the information displayedon the CT image, such as the location and orientation of the surgicalinstrument, the surgical path, and nearby anatomical cavities andstructures, the surgeon must first mentally transform or relate theirspatial understanding of the axial plane to the sagittal plane in orderto know which direction to navigate the surgical instrument. Thisprocess can be mentally taxing, may consume valuable time during aprocedure, and may also lead to erroneous navigation of the surgicalinstrument.

To address this, the visualization system (60) provides a framework forrelating the coordinate system and associated information to thephysical world perceived by the wearer of the HMD (100). Such associatedinformation may include, for example, CT images, configured surgicalpaths, real time surgical instrument tracking, real time patienttracking, configured points of interest indicating areas that should beinvestigated or avoided, and other similar information that may becorrelated to a coordinate system for IGS navigation, which may becollectively referred to herein as correlated datasets.

Once related, these correlated datasets can then be displayed via thedisplay (104) so that they are available when directly looking at thepatient instead of only being displayed on the display screen (16) oranother nearby display. In addition to reducing the need to refer toexternal displays, such an implementation allows the wearer of the HMD(100) to browse or navigate the correlated datasets by changing theirperspective relative to the patient, instead of using a mouse orkeyboard to step through image slices, switch between viewable planes,or rotate 3-D models. For example, in the case of a tracked surgicalinstrument location and surgical path, the surgeon may be able to view arendered overlay of the instrument location and surgical path within thepatient's head from different perspectives by moving and observing fromdifferent angles, rather than being confined to stepping through CTimage slices and between CT image planes using a mouse or keyboardinterface.

As an exemplary implementation of the above, FIG. 5 shows a set of steps(300) that may be performed with the visualization system (60) to renderand display correlated datasets via the HMD (100) or another viewingdevice, while FIGS. 6A-6C and FIGS. 7A-7C show exemplary interfaces thatmay be displayed or viewed with the visualization system (60). Afterpositioning a patient for a procedure, the patient may be registered(block 302) and, in implementations including the patient trackingassembly (50), tracked within the IGS navigation coordinate system.Registration of the patient may include using a registration probe orother device to provide the coordinate system with a plurality oflocations that correspond to patient anatomy, may include placing,calibrating, and using the patient tracking assembly (50), or both, ormay include other registration techniques. Registration (block 302) mayalso include registering and tracking other devices and instruments,such as the guidewire (40) and other trackable surgical instruments.Registration (block 302) may also include registering and tracking theHMD (100), where it is capable of being positionally tracked by the IGSnavigation system (10).

The IGS navigation system (10) may also receive (block 304) one or morecorrelated datasets that are associated with the patient and procedure,which may include pre-operative images of the patient anatomy (e.g., CT,MRI, and ultrasound image sets), pre-configured surgical plans andsurgical paths, and other pre-configured or pre-operatively captured orgenerated datasets that may be associated with the IGS navigationcoordinate system. The received (block 304) correlated datasets may alsoinclude data that is captured in real-time during the procedure and thenassociated with the coordinate system, such as position tracking dataindicating the location of the guidewire (40) and other tracked surgicalinstruments, and position tracking data indicating the location of theHMD (100).

When the HMD (100) or another device (e.g., the HHD (102)) is in usewith the visualization system (60), images may be captured (block 306)by the camera (106), as has been described. In some implementations,images will be captured (block 306) constantly based upon a configuredframerate of the camera (106), such that each subsequent image maychange slightly from the previous image based upon movements of thewearer of the HMD (100). Captured (block 306) images may be stored bythe processor and memory (114) and may be transmitted to the processor(12) or another device.

As images are captured (block 306), the visualization system (60) mayrepeatedly determine (block 308) the orientation of the HMD (100)relative to the anatomy and repeatedly determine (block 310) thedistance of the HMD (100) relative to the anatomy. These determinations(block 308, block 310) may be made continuously and independently, ormay be made for each captured (block 306) image one a one-to-one basis(e.g., where the camera (106) captures thirty images or frames persecond, the visualization system (60) would determine orientation (block308) and distance (block 310) thirty times per second, once for eachimage) or some other correspondence (e.g., the visualization system maybe configured to determine orientation (block 308) and distance (block310) once for every three captured (block 306) images), as will beapparent to those skilled in the art in light of this disclosure.

The orientation (308) and distance (310) may be determined in varyingways, as has already been described. For example, in someimplementations, each of the HMD (100) and the patient head (H) may bepositionally and orientationally tracked by the IGS navigation system(10) and the distance and orientation may be determined using the IGSnavigation coordinate system. In some implementations, the orientationand/or distance may be determined using the perspective sensor (110) ofthe HMD (100).

In some implementations, the orientation and/or distance may bedetermined using image analysis of a captured (block 306) image toidentify a particular object (e.g., an optical fiducial) or patientanatomy (e.g., an eye). For example, particularly in the case of anoptical fiducial having a predictable size, shape, and othercharacteristics, image analysis of an image containing the opticalfiducial can indicate the distance and perspective from which theoptical fiducial is viewed. With reference to FIG. 6A, the patienttracking assembly (200) can be seen placed on the head (H) of a patient.One or more optical fiducials (230, 232) may be placed on the patienttracking assembly (200) or elsewhere, as may be desirable.

The appearance of the optical fiducial (230) in an image provides anindication of the perspective from which the image was captured (e.g.,the optical fiducial (230) may appear as a circle when viewed as shownin FIG. 6A, but may appear as an oval as a viewer moves to the left orright) as well as the distance (e.g., the optical fiducial might have adiameter of 2 cm. Where the optical fiducial (230) has an asymmetricalshape of sufficient complexity, or where the surface of the opticalfiducial (230) has a pattern or other visible characteristic ofsufficient complexity, a single fiducial may provide enough informationto fully determine orientation and distance.

Additionally, where several optical fiducials (230, 232) are used, suchas is shown in FIG. 6A, the apparent positions of each fiducial relativeto the others may be used as an indication of orientation. As such, itmay be advantageous to place two or more optical fiducials on thetracking assembly (200) or the head (H). In some implementations, one ormore fiducials may be integrated onto the surface of the trackingassembly (200) at the time of manufacture, which may advantageouslyindicate a static and known positioning and distance from each otherwhich may be used to aid in subsequent fiducial based orientation anddistance determinations.

As has been described, implementations may vary in the particularapproach that is taken for determining the orientation (block 308) andthe distance (block 310), and while some implementations may relyentirely on tracking each object of interest within the IGS navigationcoordinate system, others may combine such tracking with image analysisof optical fiducials or other techniques in order to improve accuracy,performance, or other characteristics of the results. As such, it shouldbe understood that various combinations of the disclosed methods andothers exist and will provide varying advantages for determining theorientation (block 308) and the distance (block 310), and suchcombinations will be apparent to those skilled in the art based on thisdisclosure.

Once determined, the distance and orientation relative to the viewedanatomy can then be used to transform a correlated dataset so that itmay be displayed via the display (104) as a rendered overlay of theviewed anatomy, or displayed via another device as a rendered overlay ofa captured image. Correlated dataset transformations may include, forexample, transforming (block 312) a surgical path to match the scale andperspective of the captured (block 306) image, transforming (block 314)a CT image or other image type to match the scale and perspective of thecaptured (block 306) image, transforming (block 315) the trackedlocation of a surgical tool distal tip to match the scale andperspective of the captured (block 306) image, and othertransformations. While they are discussed within the context of FIG. 5,it should be understood that a surgical path, CT image, and trackedsurgical tool are not required to be present within the received (block304) correlated datasets.

Transformation of correlated datasets will vary depending upon theparticular data represented in a correlated dataset. For example, withreference to FIG. 8, where the system is configured to overlay some orall of a CT image slice on a patient's head (H), the distance (506)between a perspective point (502) (e.g., the lens of the camera (106))and a viewed point (504) (e.g., the first point, voxel, or other objectwithin the coordinate system intersected by the optical axis of thecamera (106)) may be used to determine and apply a scaling factor totransform and control the scale at which the CT image slice is displayedvia the display (104). As the wearer of the HMD (100) moves toward thehead (H), the distance (506) will reduce and the scale of the displayedCT image slice will increase. Similarly, movement away from the head (H)will increase the distance (506) and reduce the scale of the displayedCT image slice. In this manner, the scaling factor can be configured toprovide, based on the distance (506), an appropriately sized renderingof the CT image that corresponds to the perceived size of the head (H)at that distance.

Continuing the above example, the position of the perspective point(502) relative to the viewed point (504), within a three dimensionalcoordinate system (501), may be used to determine an appropriate CTimage slice to render, and may be used to transform an appropriate CTimage slice so that it may be overlaid on the head (H). As an example,in some implementations a CT image slice of the head (H) may be selectedand rendered as an overlay depending upon the perspective from which thehead (H) is viewed, such that a perspective above the head (H) mightshow an axial view, a perspective from in front of the head (H) mightshow a coronal view, and a perspective from the side of the head mightshow a sagittal view, and views may be switched automatically as thewearer of the HMD (100) moves between perspectives.

As another example, in some implementations a CT image slice may betransformed to create a new image having an appearance of that of thetwo-dimensional input image as if it were fixed in place and viewed froma different perspective (e.g., a two-dimensional image perspectivetransformation). In this manner, a two-dimensional CT image slicedisplayed on the head (H) might be perspective transformed as the wearerof the HMD (100) moves between perspectives.

As yet another transformation example, in the case of a correlateddataset containing a surgical path the coordinates of the surgical pathmay be rendered and overlaid on the head (H) as a set of points, a line,or a dotted line. The distance (506) may be used to transform the scaleof the surgical path so that, when overlaid upon the head (H), eachcoordinate of the surgical path is appropriately positioned relative tothe head (H). The position of the perspective point (502) relative tothe viewed point (504) within the coordinate system (501) may be used totransform the surgical path as the wearer of the (HMD) moves betweenperspectives. Such a transformation may be performed as a perspectivetransformation, as described above, or may be performed using otherthree dimensional rotational and depth transformations, as will beapparent to those skilled in the art based on the disclosure herein.

After each correlated dataset is transformed (e.g., scaled, perspectivetransformed, or otherwise), they may be rendered or displayed (block316) on one or more viewing devices, which may include displayingrendered markers as overlays via the display (104), displaying renderedmarkers as overlays on a captured image via the HHD (101) or anotherdisplay, or both. For a user wearing the HMD (100), the rendered markersmay appear to overlay objects within their field of view (e.g., thepatient's head or other anatomy) that are viewed through the transparentdisplay (104). For users of the HHD (101) or viewers of a wall mounteddisplay or other device, a captured image may be displayed with therendered markings overlaid thereon. As has been described, the steps ofcapturing images, determining perspective, and transforming correlateddatasets may be repeated continuously as images are captured so thatusers may move and look around a procedure area as normal whilereceiving continuous updates of overlaid information.

For example, where either a movement of the viewed anatomy occurs (block318) or a movement of the HMD (100), HHD (101), or other viewing deviceoccurs (block 320), the next captured (block 306) image will bereceived, and the orientation (block 308) and distance (block 310) willbe redetermined. The newly determined orientation and distance will thenbe used for one or more transformations, and the newly produced overlayswill account for any movements or changes that have occurred since theprior image.

FIGS. 6A-6C show an example of an interface (400) that includes arendered and transformed surgical path, such as may be displayed via thedisplay (104) and overlaid upon a directly viewed patient's head (H), orsuch as may be overlaid upon a captured image of the head (H) anddisplayed. In FIG. 6A, the head (H) is viewed from the front (e.g., aview of the coronal plane), and rendered markers have been overlaid viathe display (104) or directly onto a captured image. The renderedmarkers include a surgical path (402) indicating the path that asurgical tool should follow, a target marker (404) indicating adestination or portion of anatomy involved in the procedure, and a toolmarker (406) indicating the current location of a tracked surgicalinstrument. The scale of the markers has been determined based upon thedetermined distance (310), such that the surgical path (402) accuratelyoverlays the patient's head (H) and appears as it would were it to beviewed directly on a coronal plane CT image slice rather than as anoverlay rendered via the display (104) or another device. The positionsand orientations of the markers have also been determined based upon thedetermined orientation (308).

FIG. 6B shows the head (H) viewed from the side (e.g., a view of thesagittal plane). Each marker has been rendered in a new position andorientation based upon the change in perspective. For example, in FIG.6A the surgical path (402) is seen to traverse along the coronal planeof the head (H). The coronal plane is not visible in FIG. 6B, and thesurgical path (402) can instead be seen traversing along the sagittalplane. Similarly, the position and orientation of the target marker(404) and tool marker (406) are each changed from FIG. 6A, and the depthat they are positioned within the head (H) can be determined. FIG. 6Cshows the head (H) viewed from a third perspective, such as might beviewed when a viewer moves between the view of FIG. 6A and the view ofFIG. 6B. As the viewer moves between the views of FIGS. 6A and 6B, or toother perspectives from which the head (H) may be viewed, each of themarkers may be updated and newly rendered to provide additionalinformation and context without being limited to only those perspectivesoffered by a CT image set, and without needing to look away from thehead (H) to the display screen (16) or another display, which may bedistracting and disorienting as has been described.

As another example of an interface that may be provided by thevisualization system (60), FIG. 7A shows an image (430) selected from aCT image set, and FIGS. 7B-7D show an example of an interface (440) thatincludes a rendered overlay of the image (430) and the surgical path(402), such as may be displayed via the display (104) and overlaid upona directly viewed patient's head (H), or such as may be overlaid upon acaptured image of the head (H) and displayed. The image (430), thesurgical path (402), and the target marker (404) may be received (304)as correlated datasets prior the procedure.

In FIG. 7B, the head (H) is viewed from the same perspective as that ofFIG. 6A. The image (430) is also overlaid upon the head (H) after havingbeen transformed so that the scale of the image (430) corresponds to theperceived size of the head (H) (e.g., as opposed to being too large suchthat the image (430) extends beyond the head (H)). The tool marker(406), surgical path (402), and target marker (404) may also be overlaidand visible with the CT image (430), and each may be rendered at varyingdegrees of transparency to either completely obfuscate direct viewing ofthe head (H) through the renderings, or to allow some degree oftransparency. While varying implementations of the visualization system(60) may provide interfaces such as shown in FIGS. 6A-6C or FIGS. 7B-7D,some implementations may provide both, and may, for example, allowcertain layers to be enabled or disabled as desirable. For example, thismay include enabling or disabling the overlay of the image (430) overthe head (H) as may be desired.

FIGS. 7C and 7D each show the head (H) from different perspectives,matching those shown in FIGS. 6B and 6C. The image (430) is simulated asa shaded area, and the other rendered markings are each transformed toaccount for the altered perspective. As has been described, FIG. 7B mayshow a substantially unaltered CT image of the coronal plane as theimage (430), while FIG. 7C may show a substantially unaltered CT imageof the sagittal plane as the image (430). In some implementations, FIG.7D may show a transformed CT image from the coronal or sagittal plane,depending upon whether the viewer's perspective is closer to the coronalplane or the sagittal plane.

It should be understood that the interfaces of FIGS. 6A-7C arerepresentative of interfaces that may be displayed when correlateddatasets are rendered and overlaid on objects that are visible throughthe transparent portions of the display (104). For example, withreference to FIG. 6A, the head (H) is not rendered via the display (104)but is a visible part of the augmented reality interface because it isbeing viewed through the transparent portion of the display (104). Theinterfaces of FIGS. 6A-7C are also representative of interfaces that maybe displayed when correlated datasets are rendered and overlaid onimages captured via the camera (106), such as may be displayed via thedisplay screen (16), or may be displayed via the HHD (101). With suchinterfaces, the head (H) would also be rendered on the display based onthe captured image.

III. Exemplary Combinations

The following examples relate to various non-exhaustive ways in whichthe teachings herein may be combined or applied. It should be understoodthat the following examples are not intended to restrict the coverage ofany claims that may be presented at any time in this application or insubsequent filings of this application. No disclaimer is intended. Thefollowing examples are being provided for nothing more than merelyillustrative purposes. It is contemplated that the various teachingsherein may be arranged and applied in numerous other ways. It is alsocontemplated that some variations may omit certain features referred toin the below examples. Therefore, none of the aspects or featuresreferred to below should be deemed critical unless otherwise explicitlyindicated as such at a later date by the inventors or by a successor ininterest to the inventors. If any claims are presented in thisapplication or in subsequent filings related to this application thatinclude additional features beyond those referred to below, thoseadditional features shall not be presumed to have been added for anyreason relating to patentability.

EXAMPLE 1

A system for ENT visualization comprising: (a) an image guided surgery(“IGS”) navigation system operable to: (i) maintain a coordinate systemcorresponding to a tracked area, (ii) track one or more position sensorswith the coordinate system, and (iii) register the location of a patientanatomy with the coordinate system; (b) a head mounted display (“HMD”)comprising a wearable frame and a display, wherein the display ispositioned on the wearable frame to be intersected by a neutral opticalaxis of the wearer; (c) a sensor operable to produce a set ofperspective data indicating the perspective of the HMD relative to thepatient anatomy; and (d) a processor; wherein the processor isconfigured to: (i) receive one or more correlated datasets, wherein eachof the one or more correlated datasets comprise data associated with thecoordinate system, (ii) while the patient anatomy is viewed from apresent perspective, determine an orientation of the neutral opticalaxis relative to a viewed point on the patient anatomy based on the setof perspective data, (iii) determine a distance between an origin of theneutral optical axis and the viewed point based on the set ofperspective data, (iv) transform the one or more correlated datasetsbased on the orientation and the distance to produce an overlay imagethat corresponds to the patient anatomy at the present perspective, and(v) render the overlay image via the display.

EXAMPLE 2

The system of example 1, wherein the one or more correlated datasetscomprise: (i) a surgical path indicating a planned route of a surgicalinstrument within the patient anatomy, and (ii) a distal tip locationindicating the current location of a distal tip associated with aposition sensor of the one or more position sensors.

EXAMPLE 3

The system of example 2, wherein the one or more correlated datasetsfurther comprise an image slice selected from a set of preoperativeimages of the patient anatomy.

EXAMPLE 4

The system of any one or more of examples 1 through 2, furthercomprising a patient tracking assembly positioned on the patientanatomy, wherein: (i) the patient tracking assembly comprises a positionsensor of the one or more position sensors, and (ii) the IGS navigationsystem is configured to update the location of the patient anatomy withthe coordinate system based on movements of the position sensor.

EXAMPLE 5

The system of example 4, wherein the sensor is mounted on the HMD and isone of the one or more position sensors, and wherein the processor isfurther configured to, when determining the orientation of the neutraloptical axis relative to the viewed point: (i) determine an orientationof the neutral optical axis based on the coordinate system, (ii)determine an orientation of the patient anatomy based on the coordinatesystem, and (iii) correlate the orientation of the neutral optical axisand the orientation of the patient anatomy based on the coordinatesystem.

EXAMPLE 6

The system of any one or more of examples 4 through 5, wherein thesensor is mounted on the HMD and is one of the one or more positionsensors, and wherein the processor is further configured to, whendetermining the distance between the origin and the viewed point: (i)determine a location of the origin based on the coordinate system, (ii)determine a location of the patient anatomy based on the coordinatesystem, and (iii) determine the distance between the location of theorigin and the location of the patient anatomy based on the coordinatesystem.

EXAMPLE 7

The system of any one or more of examples 1 through 6, wherein theprocessor is further configured to, when transforming the one or morecorrelated datasets: (i) determine a scaling factor based on thedistance between the origin and the viewed point, (ii) determine aperspective transform based on the orientation of the neutral opticalaxis relative to the viewed point, and (iii) produce the overlay imagecomprising a scale determined by the scaling factor and a perspectivedetermined by the perspective transformation.

EXAMPLE 8

The system of example 7, wherein the display comprises a transparentscreen, and wherein the processor is further configured to render theoverlay image on the transparent screen such that the overlay imageappears on a directly viewed portion of the patient anatomy.

EXAMPLE 9

The system of any one or more of examples 1 through 8, wherein theprocessor is further configured to: (i) repeatedly redetermine thedistance and the orientation, and (ii) update and render the overlayimage via the display as the distance and orientation change.

EXAMPLE 10

The system of any one or more of examples 1 through 9, wherein thesensor comprises a camera positioned on the wearable frame and having anoptical axis that is parallel to and statically offset from the neutraloptical axis, and wherein the processor is further configured to, whendetermining the orientation and the distance: (i) store an objectdataset that indicates one or more visible characteristics of an objectand indicates a relationship between the object and the viewed pointwithin the coordinate system, (ii) receive an image from the camera thatcontains the object, and (iii) determine the orientation and thedistance based on the object dataset and the image.

EXAMPLE 11

The system of example 10, further comprising one or more opticalfiducials positioned proximately to the patient anatomy, wherein theobject described by the object dataset is the one or more opticalfiducials.

EXAMPLE 12

The system of example 11, further comprising a patient tracking assemblypositioned on the patient anatomy, wherein: (i) the patient trackingassembly comprises a position sensor of the one or more positionsensors, (ii) the IGS navigation system is configured to update thelocation of the patient anatomy with the coordinate system based onmovements of the position sensor, and (iii) the one or more opticalfiducials are positioned on a surface of the patient tracking assembly.

EXAMPLE 13

The system of any one or more of examples 1 through 12, wherein thesensor comprises a camera positioned on the wearable frame and having anoptical axis that is parallel to and statically offset from the neutraloptical axis, and wherein the processor is further configured to, inaddition to rendering the overlay image via the display: (i) receive animage from the camera and correlate the image to the overlay image basedon a known static offset value, (ii) add the overlay image to the imageto produce an augmented image, and (iii) display the augmented image onone or more displays other than the display.

EXAMPLE 14

The system of any one or more of examples 1 through 13, wherein: (i) theprocessor comprises one or more of a first processor of the IGSnavigation system and a second processor of the HMD, (ii) the sensorcomprises one or more of a gyroscope sensor coupled with the HMD, acamera mounted on the HMD, and a position sensor of the one or moreposition sensors mounted on the HMD, and (iii) the set of perspectivedata comprises one or more of a set of gyroscopic orientation data, aset of position coordinates associated with the coordinate system, a setof orientation coordinates associated with the coordinate system, and animage captured by the camera.

EXAMPLE 15

A method comprising: (a) configuring an image guided surgery (“IGS”)navigation system to: (i) maintain a coordinate system corresponding toa tracked area, and (ii) track one or more position sensors with thecoordinate system; (b) registering the location of a patient anatomywith the coordinate system; (c) receiving one or more correlateddatasets, wherein each of the one or more correlated datasets comprisedata associated with the coordinate system; (d) viewing the patientanatomy from a present perspective while wearing a head mounted display(“HMD”) comprising a wearable frame and a display, wherein the displayis positioned on the wearable frame to be intersected by a neutraloptical axis of the wearer; (e) determining an orientation of theneutral optical axis relative to a viewed point on the patient anatomybased on a set of perspective data received from a sensor of the HMD;(f) determining a distance between an origin of the neutral optical axisand the viewed point based on the set of perspective data; (g)transforming the one or more correlated datasets based on theorientation and the distance to produce an overlay image thatcorresponds to the patient anatomy at the present perspective; and (h)rendering the overlay image via the display.

EXAMPLE 16

The method of example 15, wherein receiving the one or more correlateddatasets comprises: (i) receiving a surgical path indicating a plannedroute of a surgical instrument within the patient anatomy, (ii)receiving a distal tip location indicating the current location of adistal tip associated with a position sensor of the one or more positionsensors, and (iii) receiving an image slice selected from a set ofpreoperative images of the patient anatomy.

EXAMPLE 17

The method of any one or more of examples 15 through 16, wherein thesensor comprises a camera positioned on the wearable frame and having anoptical axis that is parallel to and statically offset from the neutraloptical axis, the method further comprising: (a) placing a patienttracking assembly on the patient anatomy, wherein the patient trackingassembly comprises: (i) a position sensor of the one or more positionsensors, and (ii) one or more optical fiducials; (b) storing an objectdataset that indicates one or more visible characteristics of the one ormore optical fiducials and indicates a relationship between the one ormore optical fiducials and the viewed point within the coordinatesystem; (c) receiving an image from the camera that contains the one ormore optical fiducials; and (d) determining the orientation and thedistance based on the object dataset and the image.

EXAMPLE 18

The method of example 17, wherein the sensor comprises a second positionsensor of the one or more position sensors, further comprisingdetermining the orientation and the distance based on the objectdataset, the image, and the coordinate system.

EXAMPLE 19

A system for ENT visualization comprising: (a) an image guided surgery(“IGS”) navigation system operable to: (i) maintain a coordinate systemcorresponding to a tracked area, (ii) track one or more position sensorswith the coordinate system, and (iii) register the location of a patientanatomy with the coordinate system; (b) a hand-held display (“HHD”)comprising a camera and a display; (c) a sensor operable to produce aset of perspective data indicating the perspective of the HHD relativeto the patient anatomy; and (d) a processor; wherein the processor isconfigured to: (i) receive one or more correlated datasets, wherein eachof the one or more correlated datasets comprise data associated with thecoordinate system, (ii) while the HHD is directed at the patient anatomyfrom a present perspective, receive an image from the camera, (iii)determine an orientation of an optical axis of the camera relative to aviewed point on the patient anatomy based on the set of perspectivedata, (iv) determine a distance between an origin of the optical axisand the viewed point based on the set of perspective data, (v) transformthe one or more correlated datasets based on the orientation and thedistance to produce an overlay image that corresponds to the patientanatomy in the image, and (vi) display an augmented image via thedisplay based on the overlay image and the image.

EXAMPLE 20

The system of example 19, wherein: (i) the sensor is a position sensorof the one or more position sensors, (ii) a second position sensor ofthe one or more positions sensors is positioned on the patient anatomy,and (iii) the one or more correlated datasets comprise: (A) a surgicalpath indicating a planned route of a surgical instrument within thepatient anatomy, and (B) a distal tip location indicating the currentlocation of a distal tip associated with a third position sensor of theone or more position sensors.

IV. Miscellaneous

It should be understood that any one or more of the teachings,expressions, embodiments, examples, etc. described herein may becombined with any one or more of the other teachings, expressions,embodiments, examples, etc. that are described herein. Theabove-described teachings, expressions, embodiments, examples, etc.should therefore not be viewed in isolation relative to each other.Various suitable ways in which the teachings herein may be combined willbe readily apparent to those skilled in the art in view of the teachingsherein. Such modifications and variations are intended to be includedwithin the scope of the claims.

It should be appreciated that any patent, publication, or otherdisclosure material, in whole or in part, that is said to beincorporated by reference herein is incorporated herein only to theextent that the incorporated material does not conflict with existingdefinitions, statements, or other disclosure material set forth in thisdisclosure. As such, and to the extent necessary, the disclosure asexplicitly set forth herein supersedes any conflicting materialincorporated herein by reference. Any material, or portion thereof, thatis said to be incorporated by reference herein, but which conflicts withexisting definitions, statements, or other disclosure material set forthherein will only be incorporated to the extent that no conflict arisesbetween that incorporated material and the existing disclosure material.

Versions of the devices described above may be designed to be disposedof after a single use, or they can be designed to be used multipletimes. Versions may, in either or both cases, be reconditioned for reuseafter at least one use. Reconditioning may include any combination ofthe steps of disassembly of the device, followed by cleaning orreplacement of particular pieces, and subsequent reassembly. Inparticular, some versions of the device may be disassembled, and anynumber of the particular pieces or parts of the device may beselectively replaced or removed in any combination. Upon cleaning and/orreplacement of particular parts, some versions of the device may bereassembled for subsequent use either at a reconditioning facility, orby a user immediately prior to a procedure. Those skilled in the artwill appreciate that reconditioning of a device may utilize a variety oftechniques for disassembly, cleaning/replacement, and reassembly. Use ofsuch techniques, and the resulting reconditioned device, are all withinthe scope of the present application.

By way of example only, versions described herein may be sterilizedbefore and/or after a procedure. In one sterilization technique, thedevice is placed in a closed and sealed container, such as a plastic orTYVEK bag. The container and device may then be placed in a field ofradiation that can penetrate the container, such as gamma radiation,x-rays, or high-energy electrons. The radiation may kill bacteria on thedevice and in the container. The sterilized device may then be stored inthe sterile container for later use. A device may also be sterilizedusing any other technique known in the art, including but not limited tobeta or gamma radiation, ethylene oxide, or steam.

Having shown and described various embodiments of the present invention,further adaptations of the methods and systems described herein may beaccomplished by appropriate modifications by one of ordinary skill inthe art without departing from the scope of the present invention.Several of such potential modifications have been mentioned, and otherswill be apparent to those skilled in the art. For instance, theexamples, embodiments, geometrics, materials, dimensions, ratios, steps,and the like discussed above are illustrative and are not required.Accordingly, the scope of the present invention should be considered interms of the following claims and is understood not to be limited to thedetails of structure and operation shown and described in thespecification and drawings.

I/We claim:
 1. A system for ENT visualization comprising: (a) an imageguided surgery (“IGS”) navigation system operable to: (i) maintain acoordinate system corresponding to a tracked area, (ii) track one ormore position sensors with the coordinate system, and (iii) register thelocation of a patient anatomy with the coordinate system; (b) a headmounted display (“HMD”) comprising a wearable frame and a display,wherein the display is positioned on the wearable frame to beintersected by a neutral optical axis of the wearer; (c) a sensoroperable to produce a set of perspective data indicating the perspectiveof the HMD relative to the patient anatomy; and (d) a processor; whereinthe processor is configured to: (i) receive one or more correlateddatasets, wherein each of the one or more correlated datasets comprisedata associated with the coordinate system, (ii) while the patientanatomy is viewed from a present perspective, determine an orientationof the neutral optical axis relative to a viewed point on the patientanatomy based on the set of perspective data, (iii) determine a distancebetween an origin of the neutral optical axis and the viewed point basedon the set of perspective data, (iv) transform the one or morecorrelated datasets based on the orientation and the distance to producean overlay image that corresponds to the patient anatomy at the presentperspective, and (v) render the overlay image via the display.
 2. Thesystem of claim 1, wherein the one or more correlated datasets comprise:(i) a surgical path indicating a planned route of a surgical instrumentwithin the patient anatomy, and (ii) a distal tip location indicatingthe current location of a distal tip associated with a position sensorof the one or more position sensors.
 3. The system of claim 2, whereinthe one or more correlated datasets further comprise an image sliceselected from a set of preoperative images of the patient anatomy. 4.The system of claim 1, further comprising a patient tracking assemblypositioned on the patient anatomy, wherein: (i) the patient trackingassembly comprises a position sensor of the one or more positionsensors, and (ii) the IGS navigation system is configured to update thelocation of the patient anatomy with the coordinate system based onmovements of the position sensor.
 5. The system of claim 4, wherein thesensor is mounted on the HMD and is one of the one or more positionsensors, and wherein the processor is further configured to, whendetermining the orientation of the neutral optical axis relative to theviewed point: (i) determine an orientation of the neutral optical axisbased on the coordinate system, (ii) determine an orientation of thepatient anatomy based on the coordinate system, and (iii) correlate theorientation of the neutral optical axis and the orientation of thepatient anatomy based on the coordinate system.
 6. The system of claim4, wherein the sensor is mounted on the HMD and is one of the one ormore position sensors, and wherein the processor is further configuredto, when determining the distance between the origin and the viewedpoint: (i) determine a location of the origin based on the coordinatesystem, (ii) determine a location of the patient anatomy based on thecoordinate system, and (iii) determine the distance between the locationof the origin and the location of the patient anatomy based on thecoordinate system.
 7. The system of claim 1, wherein the processor isfurther configured to, when transforming the one or more correlateddatasets: (i) determine a scaling factor based on the distance betweenthe origin and the viewed point, (ii) determine a perspective transformbased on the orientation of the neutral optical axis relative to theviewed point, and (iii) produce the overlay image comprising a scaledetermined by the scaling factor and a perspective determined by theperspective transformation.
 8. The system of claim 7, wherein thedisplay comprises a transparent screen, and wherein the processor isfurther configured to render the overlay image on the transparent screensuch that the overlay image appears on a directly viewed portion of thepatient anatomy.
 9. The system of claim 1, wherein the processor isfurther configured to: (i) repeatedly redetermine the distance and theorientation, and (ii) update and render the overlay image via thedisplay as the distance and orientation change.
 10. The system of claim1, wherein the sensor comprises a camera positioned on the wearableframe and having an optical axis that is parallel to and staticallyoffset from the neutral optical axis, and wherein the processor isfurther configured to, when determining the orientation and thedistance: (i) store an object dataset that indicates one or more visiblecharacteristics of an object and indicates a relationship between theobject and the viewed point within the coordinate system, (ii) receivean image from the camera that contains the object, and (iii) determinethe orientation and the distance based on the object dataset and theimage.
 11. The system of claim 10, further comprising one or moreoptical fiducials positioned proximately to the patient anatomy, whereinthe object described by the object dataset is the one or more opticalfiducials.
 12. The system of claim 11, further comprising a patienttracking assembly positioned on the patient anatomy, wherein: (i) thepatient tracking assembly comprises a position sensor of the one or moreposition sensors, (ii) the IGS navigation system is configured to updatethe location of the patient anatomy with the coordinate system based onmovements of the position sensor, and (iii) the one or more opticalfiducials are positioned on a surface of the patient tracking assembly.13. The system of claim 1, wherein the sensor comprises a camerapositioned on the wearable frame and having an optical axis that isparallel to and statically offset from the neutral optical axis, andwherein the processor is further configured to, in addition to renderingthe overlay image via the display: (i) receive an image from the cameraand correlate the image to the overlay image based on a known staticoffset value, (ii) add the overlay image to the image to produce anaugmented image, and (iii) display the augmented image on one or moredisplays other than the display.
 14. The system of claim 1, wherein: (i)the processor comprises one or more of a first processor of the IGSnavigation system and a second processor of the HMD, (ii) the sensorcomprises one or more of a gyroscope sensor coupled with the HMD, acamera mounted on the HMD, and a position sensor of the one or moreposition sensors mounted on the HMD, and (iii) the set of perspectivedata comprises one or more of a set of gyroscopic orientation data, aset of position coordinates associated with the coordinate system, a setof orientation coordinates associated with the coordinate system, and animage captured by the camera.
 15. A method comprising: (a) configuringan image guided surgery (“IGS”) navigation system to: (i) maintain acoordinate system corresponding to a tracked area, and (ii) track one ormore position sensors with the coordinate system; (b) registering thelocation of a patient anatomy with the coordinate system; (c) receivingone or more correlated datasets, wherein each of the one or morecorrelated datasets comprise data associated with the coordinate system;(d) viewing the patient anatomy from a present perspective while wearinga head mounted display (“HMD”) comprising a wearable frame and adisplay, wherein the display is positioned on the wearable frame to beintersected by a neutral optical axis of the wearer; (e) determining anorientation of the neutral optical axis relative to a viewed point onthe patient anatomy based on a set of perspective data received from asensor of the HMD; (f) determining a distance between an origin of theneutral optical axis and the viewed point based on the set ofperspective data; (g) transforming the one or more correlated datasetsbased on the orientation and the distance to produce an overlay imagethat corresponds to the patient anatomy at the present perspective; and(h) rendering the overlay image via the display.
 16. The method of claim15, wherein receiving the one or more correlated datasets comprises: (i)receiving a surgical path indicating a planned route of a surgicalinstrument within the patient anatomy, (ii) receiving a distal tiplocation indicating the current location of a distal tip associated witha position sensor of the one or more position sensors, and (iii)receiving an image slice selected from a set of preoperative images ofthe patient anatomy.
 17. The method of claim 15, wherein the sensorcomprises a camera positioned on the wearable frame and having anoptical axis that is parallel to and statically offset from the neutraloptical axis, the method further comprising: (a) placing a patienttracking assembly on the patient anatomy, wherein the patient trackingassembly comprises: (i) a position sensor of the one or more positionsensors, and (ii) one or more optical fiducials; (b) storing an objectdataset that indicates one or more visible characteristics of the one ormore optical fiducials and indicates a relationship between the one ormore optical fiducials and the viewed point within the coordinatesystem; (c) receiving an image from the camera that contains the one ormore optical fiducials; and (d) determining the orientation and thedistance based on the object dataset and the image.
 18. The method ofclaim 17, wherein the sensor comprises a second position sensor of theone or more position sensors, further comprising determining theorientation and the distance based on the object dataset, the image, andthe coordinate system.
 19. A system for ENT visualization comprising:(a) an image guided surgery (“IGS”) navigation system operable to: (i)maintain a coordinate system corresponding to a tracked area, (ii) trackone or more position sensors with the coordinate system, and (iii)register the location of a patient anatomy with the coordinate system;(b) a hand-held display (“HHD”) comprising a camera and a display; (c) asensor operable to produce a set of perspective data indicating theperspective of the HHD relative to the patient anatomy; and (d) aprocessor; wherein the processor is configured to: (i) receive one ormore correlated datasets, wherein each of the one or more correlateddatasets comprise data associated with the coordinate system, (ii) whilethe HHD is directed at the patient anatomy from a present perspective,receive an image from the camera, (iii) determine an orientation of anoptical axis of the camera relative to a viewed point on the patientanatomy based on the set of perspective data, (iv) determine a distancebetween an origin of the optical axis and the viewed point based on theset of perspective data, (v) transform the one or more correlateddatasets based on the orientation and the distance to produce an overlayimage that corresponds to the patient anatomy in the image, and (vi)display an augmented image via the display based on the overlay imageand the image.
 20. The system of claim 19, wherein: (i) the sensor is aposition sensor of the one or more position sensors, (ii) a secondposition sensor of the one or more positions sensors is positioned onthe patient anatomy, and (iii) the one or more correlated datasetscomprise: (A) a surgical path indicating a planned route of a surgicalinstrument within the patient anatomy, and (B) a distal tip locationindicating the current location of a distal tip associated with a thirdposition sensor of the one or more position sensors.